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Abstract 

This study examines the reliability and validity of the Application of Cognitive Functions 
Scale (ACFS) Behavior Observation Rating Scale (BORS). The study involved 
observational ratings of participants’ (25) who were enrolled in a preschool for children 
with developmental disabilities. Validity was investigated by how well the researcher’s 
ratings during a sample classroom activity and speech therapy session correlated with the 
teachers’ and speech therapists’ ratings of the children's general behavior in class and in 
speech therapy, respectively. Reliability was examined by determining inter-rater 
agreement between scores of the scale's author and the researcher by reviewing 
videotaped samples of preschool children. In addition, reliability was examined by 
investigating the consistency of the children's behavior across the two varying situations, 
of speech session and a general classroom session. 
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Introduction 

Dynamic assessment was originally proposed by the Soviet psychologist L.S. 
Vygotsky (Berk & Winsler, 1995) as an alternative to traditional psychometric assessment. 
Vygotsky criticized traditional approaches for lack of treatment validity and limited 
information about the child's development (Berk & Winsler, 1995). 

The uniqueness of a dynamic approach to assessment can be best characterized 
by the interaction between the examiner and the examinee. In contrast to traditional 
assessment where the examiner is confined to specific rules and guidelines during the 
assessment, according to the model used in this study, the examiner administering a 
dynamic approach functions as a facilitator, interacting with the child in ways that reveal 
the child's potential or emergent level of development. Due to the interactive nature of the 
procedure, the examiner has an opportunity to observe, make inferences, and remediate 
problems of cognitive processing in the child (Lidz, 1995). 

Dynamic assessment procedures characteristically follow a test-intervene - 
(mediate) post-test format that allows documentation of the child's responsiveness to 
instruction (intervention-mediation). Most of the existing procedures are appropriate for 
use with either school age or older learners. There are as yet no dynamic assessment 
procedures available for use with children younger than five years. 

Lidz and Jepsen (1997) designed the Application of Cognitive Functions Scale to 
assess higher mental functioning in children between the ages of three through five years. 
In addition to providing scores for successful solution of six tasks and responsiveness of 
the child to facilitation of each task, the ACFS also includes a Behavior Observation 
Rating Scale (BORS) for each of the tasks. The examiner rates the child's behavior on 
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seven parameters during administration of each pretest and mediation phase of the six 
activities. Since the ACFS is a relatively new procedure, information is needed to 
document its validity and reliability. This study will contribute to this need by investigating 
the validity and reliability of this subcomponent of the ACFS, the BORS. 

Literature Review 

Many standardized procedures provide an informal checklist or rating scale for the 
examiner to record the child's interactive behavior during the course of the administration 
of the procedure. Glutting, Oakland, and McDermott (1989) are among the few who have 
tried to formalize this process. However, their scale is appropriate for children of school 
age. The ACFS is unique in including both qualitative and quantitative scores for 
preschool children. The contribution of the qualitative aspect of the information, provided 
by the Behavior Observation Scale, requires further study. 

This literature review focuses on documentation of the behaviors incorporated in 
the BORS (Appendices A & B). The ACFS authors selected the seven parameters 
because they judged them to be observable and measurable during the administration of 
the ACFS and, primarily, because the behaviors were representative of instructional^ 
relevant descriptors of young children. These parameters include the following behaviors: 
self-regulation, persistence, frustration tolerance, flexibility, motivation, interactivity, and 
receptivity. The ratings reflect the extent to which the young child is developing into a 
thinking, reflective, metacognitive learner. 

Metacognition refers to knowledge concerning ones' own cognitive processes and 
products (Flavell, 1976a). Metacognitive strategies include those that connect new 
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information to former knowledge, planning, monitoring, and selective thinking. There is 
evidence that increases in learning have followed direct instruction in metacognitive 
strategies (Flavell, 1976a). These results suggest that direct teaching of these thinking 
strategies may be useful. It is therefore relevant to assess metacognitive characteristics 
in children since there is the potential for linkage with instruction. Learning how to learn 
and developing a repertoire of thinking processes that can be applied to solve problems is 
a major goal for education. However, very young children are only at an "emergent" level 
in development of these metacognitive abilities. Nevertheless, these capacities begin to 
emerge at approximately the age of three years in the child with typical development 
(Flavell, 1976b). 

Review of the literature related to the specific components of the Behavior 
Observation Rating Scale follows: 

Self-regulation 

As defined by the BORS (see Appendixes A and B), self-regulation means 
regulating and maintaining attention and controlling impulsivity. Self-regulated learning is 
pivotal to student achievement; the most effective learners are self-regulating (Butler & 
Winne, 1995). In the academic context, self-regulation includes setting goals for 
increased knowledge, considering strategies for achieving those goals without wasting 
time or energy, and regularly monitoring progress. Self-regulated students know their own 
capacities, beliefs, and knowledge and can adjust their goal -setting and motivation and 
even create new strategies to improve their progress. 
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Feedback, however, is essential to self-regulated learning, but self-regulated 
learners often create their own internal feedback methods, as well as seek feedback from 
external sources (Butler & Winne, 1995). Self-regulated learning might follow these 
stages: 1) learners begin with knowledge and beliefs, strategies, and multiple motivations; 
2) they progress to setting goals and adapting tactics and strategies for reaching those 
goals; 3) Internal feedback and self-monitoring advanced achievement of goals or is 
adjusted if progress seems unsatisfactory and 4) results are subjected to external 
feedback (Butler & Winne 1995; Zimmerman, 1990). 

It is important to understand that the self-regulated learner is not necessarily 
discouraged either by internal or external negative feedback; he merely re-creates his 
strategies and / or adjusts his goal setting or time frame for achieving those goals (Butler 
& Winne, 1995; Zimmerman, 1990). In effect, the feedback loop itself is a self-oriented 
process in which the student engages in a cycle of receiving feedback from himself or 
external sources, altering his self-perceptions, changing behaviors, or changing a learning 
strategy, and repeating the cycle until the goal is achieved (Zimmerman, 1990). 

The feedback itself that is so pivotal to self-regulated learning might be knowledge 
of results, task validity feedback, cognitive validity feedback, or functional validity (Butler 
& Winne, 1995). Task validity feedback defines the task; cognitive validity feedback 
describes the learner's understanding of the relationship between a step towards learning 
and the actual knowledge; and functional validity feedback describes the actual 
relationship between the student's own estimates of achievement and actual performance. 

Self-regulated learning has a long-range quality. It does not depend on either self- 
efficacy or immediate goal achievement but, instead, on a student's awareness of the 
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relationship between strategy and response and, then, learning outcomes and the use of 
and adjustment of strategies to achieve academic goals (Zimmerman, 1990). Put more 
simply, the learner who cannot self-regulate might set a goal and work towards it, but if he 
does not achieve it quickly, he will give up and abandon the task or wait for further 
external guidance. The self-regulated learner repeatedly reviews his progress and 
repeatedly adjusts his strategies until he perceives progress, with or without the external 
feedback (Zimmerman, 1990). Zimmerman (1990) also notes that motivation is a critical 
part of self-regulation in that learning activities are consistently self-initiated. 

Although some students are inherently self-regulating, many have to be taught self- 
regulating strategies. Sawyer, Graham, and Harris (1992) observed that effective strategy 
instruction included target strategies, information about the use and significance of 
strategies, and the development of self-regulation skills for monitoring and maintaining 
strategies. In their study of teaching writing strategies to fifth and sixth grade learning 
disabled students, these authors found that teaching self-efficacy, as well as performance 
and components analysis strategies improved students' writing skills and willingness to 
apply strategies to achieve their writing goals. 

Persistence 

On the BORS (see Appendices A and B), persistence is defined as completion of a 
task or activity without seeking to terminate it. 

In a learning context, persistence can be measured by the total time or numbers of 
attempts at which an individual works at a given task before turning to some alternative, 




10 



The ACFS Behavior Rating Scale 10 



especially assuming the task is difficult or challenging and the individual has all the time 
he wants to complete it (Feather, 1962). 

Persistence can be a character trait or it might be resistance to extinction or it can 
be a motivational phenomenon (Feather, 1962). In the academic setting, teachers will find 
that some students are naturally more persistent; they might even be labeled stubborn. 
Others will stay with a task if encouraged by the teacher (or peer approval); still others will 
give up easily with or without encouragement if confronting something they cannot master 
quickly. Some students will refuse to give up unwanted behaviors, even if nothing 
reinforced these behaviors except the students' own will or even if they are punished for 
those behaviors (Feather, 1962; Krantz& Scarth, 1979). 

Persistence can be encouraged and developed, especially in young children and 
even among handicapped children (Karnes, Johnson & Beauchamp, 1989; Krantz & 
Scarth, 1979). Krantz and Scarth (1979) noted that teacher behaviors, such as verbal 
reinforcement, prompting, and proximity tended to reinforce persistence on assigned tasks 
in preschool children. Karnes et al. (1989) found that teaching problem-solving strategies 
to preschool handicapped children with a view towards developing their fine motor skills 
not only increased their willingness to stay on-task but helped them become more 
systematic in problem-solving tasks. As the children began to develop these problem- 
solving strategies and become more systematic and organized in their work, their own 
success encouraged persistence. 

Krantz and Scarth (1979) also noted that a teacher's irrelevant comments prior to 
children's beginning an assignment had the effect of distracting them from staying on-task, 
whereas teachers whose students demonstrated high task persistence were those who 
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used praise freely and minimized criticism. Hamilton and Gordon (1978) observed that 
students varied within themselves regarding persistence in classroom activities, while, in 
turn, teachers varied their behaviors in relation to individual children. The authors 
hypothesized that praise and staying on-task would correlate positively and that criticism 
and giving up would also correlate positively. 

However, Hamilton and Gordon (1978) found that while both praise and criticism 
correlated positively with persistence in regular classroom tasks, there was a negative 
correlation between teacher praise and children's persistence in the experimental task. In 
the experimental situation, teachers withheld both praise and criticism, which indicated 
that those children who had come to expect a lot of teachers' attention in the past gave up 
easily when it was not forthcoming. This agrees with Krantz and Scarth (1979), who 
found a correlation between praise and staying on task and between criticism and giving 
up easily. 

Frustration Tolerance 

As described by the BORS (see Appendices A and B), the definition of frustration 
tolerance is recouping and continuing with a task when the child becomes frustrated, or 
calming down and accepting redirection when upset. At any age, frustration occurs when 
an individual is prevented from reaching a desired goal, possibly because someone or 
something is in the way or possibly because there is a lack of skill or knowledge about 
how to achieve an objective (Agnew, 1984). When aspirations and expectations fall far 
apart on the individual's scale of reality, there is strain, which also occurs when 
achievements do not meet expectations. Frustration also occurs as a consequence of 
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deprivation, punishment, feelings of inadequacy, or threats to self-esteem (Fabes, 
Eisenberg, McCormick & Wilson, 1988). 

These contributing factors to frustration are the same for children as for adults; the 
main difference is that the child's aspirations, goals, and expectations necessarily differ 
from those of the adult. Young children tend to become frustrated over object deprivation. 
If thwarted in their desire to hold or play with an object, their frustration often leads them to 
physical strategies (hitting or crying) to resolve their feelings of frustration and conflict with 
the source of their frustration. Somewhat older children might also use verbal strategies 
(loud or insulting arguments) to deal with frustration, but either way, conflict over objects 
or anything else represents the child's inability to tolerate frustration (Wheeler, 1994). 

Frustration tolerance, therefore, is a critical component of socialization of young 
children, not only in the sense of developing general social skills but in the very important 
context of learning. If the child cannot learn to tolerate frustration each time an objective 
is not immediately achieved or satisfied, the child will not develop the patience or try new 
strategies or have the persistence to keep trying until his objectives are accomplished 
(Fabes & Eisenberg, 1992; Wheeler, 1994). 

If conflict due to frustration among young children were only about objects, it would 
be relatively easily resolved, if only by presenting the frustrated child with another toy or 
otherwise redirecting the child's attention to something equally appealing. The point of 
teaching children to tolerate frustration is that if the young child cannot learn to take turns 
playing with a particular toy, it is unlikely the child will learn to wait her turn in more difficult 
and critical situations, such as needing teacher's help with an academic task, or learning 
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new skills, transferring to a new school, learning to make friends, or learning to get along 
with a new teacher in the next grade (Levin, 1994; Wheeler, 1994). 

Frustration tolerance is not only a learned social skill but also a learned intellectual 
skill (Stone, 1 993). The child, who can learn to wait her turn to play with a particular toy or 
find something else to do, is the child who can learn to try a different approach to a difficult 
and challenging academic task when the first approach does not work. Frustration 
tolerance, in effect, underlies all problem-solving skills and strategies that are the basis for 
all learning. Learning itself is by definition a confrontation with the unknown, but the 
confrontation often involves frustration that must be tolerated and coped with 
constructively if the learning goal is to be achieved (Fabes & Eisenberg, 1992; Feather, 
1962; Stone, 1993). 

Moreover, because the inability to tolerate frustration often results in anger and 
disruptive behaviors among young children, a whole classroom can be prevented from 
learning by the frustration responses of one child. It is therefore critical that even the 
youngest children learn to work through their reactions to frustrations. The inability to 
tolerate frustration in a learning context is the opposite of the persistence needed to 
complete a task and achieve a goal (Fabes & Eisenberge, 1992; Feather, 1962; Stone, 
1993). 

Flexibility 

Flexibility as defined by the BORS involves trying alternative approaches or 
solutions to problems, self-correcting, or voluntarily changing an approach when a 
previous attempt is found ineffective. Although persistence and willingness to stay on task 
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for as long as it takes to complete the task are considered pivotal to learning, these 
characteristics can be self-defeating if the student's persistence only means doing the 
same thing over and over with few or no results and little or no progress towards the 
desired objective (Nelson-LeGall & Scott-Jones, 1985). 

Persistence has to be accompanied by flexibility if any learning is to occur. Merely 
staying on task can be counterproductive if the task is especially difficult and, even more 
so, if asking for help or stopping and starting over with an entirely different strategy would 
solve the problem more effectively and efficiently (Cheyne & Rubin, 1983, Kanevesky, 
1990: Nelson-LeGall & Scott-Jones, 1985). 

This does not mean that it is desirable that a student should take one look at a 
problem and immediately call for help. It does mean that after a student, even the 
youngest student, has made a concerted effort to learn how to solve a problem 
independently and has made no progress, she should be flexible enough to seek or try 
alternative approaches. Alternative approaches might include going beyond the obvious 
first solution and trying something radically different just to see what will happen, such as 
the use of an entirely different tool or device to achieve the same goal. Alternative 
solutions might also include just a different way of using the original approach (Cheyne & 
Rubin, 1983). 

As indicated on the AFCS, the lack of flexibility is shown by a student who makes 
no changes in his approach to a problem, when the first try does not work, and he seems 
to be stuck on the first attempt. Such students might simply wait passively for the teacher 
to help; this is not a valid alternative but instead is the same as doing nothing because the 
task seemed too difficult. Flexibility is shown by the student who is willing to try active, 
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goal-oriented alternative approaches or strategies to problem solving. In a learning 
situation, the student who is flexible will not only use different strategies but apply each 
strategy in different ways, using information provided by teachers or even the examples 
provided by or other students (Nelson-LeGall & Scott-Jones, 1985; Stone, 1995). 

The point is that the flexible student does not abandon the task but only 
relinquishes a particular strategy that has proven ineffective. He does not absolutely 
refuse assistance but seeks guidance or help only to the point that he can try 
independently to complete the task. The flexible student might try to adapt strategies 
learned from past problem solving and experiment with such strategies to see if they are 
effective with the present problem. The flexible student might even try strategies in a 
different way (Kanevesky, 1990; Cheyne & Rubin, 1983; Tegano, Sawyers &Moran,1989). 

At the preschool level, play, especially dramatic and constructive play, is the ideal 
opportunity for children to display flexible behaviors, because play is intrinsically motivated 
and process-oriented. Play also permits children to use divergent thinking processes and 
to research solutions to problems, as well as express their creativity. In constructive play, 
children who are flexible will invent, re-invent, design, re-design, and create and re-create 
as they experiment with tools and materials. In dramatic play, children will draw on many 
experiences and synthesize them together to form new experiences, but if the results are 
not satisfactory, the flexible child will strive for a new synthesis of past and present 
experiences (Stone, 1995; Tegano et al., 1989). 

In play, flexible children test out concepts, revising and refining them in response to 
the perceived effectiveness of solving a problem. In effect, although the young child is not 
likely to talk through his problem-solving strategies, his degree of flexibility can be 
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discerned by his willingness and ability to play through solutions to problems (Stone, 

1995; Tegano et al., 1989). 

Motivation 

The BORS defines motivation as showing positive affective response to and 
interest in the activity. 

Intrinsic motivation has been defined as participating in an activity simply out of 
curiosity, an inner need to know about something and the desire to participate in an 
activity purely for the sake of completing a task (Deci, Vellerand, Pelletier & Ran, 1 991 ). 
The intrinsically motivated student is also likely to retain the concepts learned and to feel 
confident about engaging in unfamiliar learning situations (Dev, 1997). However the 
amount of interest generated by the task also plays a role in the motivational orientation of 
the learner. An assigned task with zero interest value is less likely to motivate the student 
than is a task that arouses interest and curiosity. 

Berlyn and Frommer’s study (cited in Gottfried, 1983) found that intrinsic motivation 
was affected by the type of stimuli with which young children were presented, such as 
stories or pictures that were incongruous, that moved and that challenged the children. 
Academic intrinsic motivation was found to be correlated with academic achievement in 
students with learning disabilities (Dev, 1997). The development of skills required for 
academic achievements are strongly influenced by instructional design. If the design 
undermines student ability and skill level it can reduce motivation. Students with learning 
disabilities have shown an increase in academic learning after engaging in interesting 
tasks such as computer games and programs designed to enhance learning (Dev, 1997). 
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Also, if the assigned task is within the child's ability level, the child is likely to be 
intrinsically motivated to tackle to task. 

Teachers can enhance the intrinsic motivation of their students by allowing the 
children to feel that they are in control of their own learning to a certain degree. Intrinsic 
motivation can develop when students are encouraged to monitor and reinforce their 
progress themselves (Brophy, 1983). Another way to maintain intrinsic motivation is when 
the student is given the opportunity to feel competent by learning through discovery 
(Adelman & Chaney, 1982). Some studies suggest that rewards have an undermining 
effect on intrinsic motivation. For example, Gottfried (1983) points out that preschool 
children who expected and received an award for engaging in an intrinsically interesting 
activity subsequently showed less interest in playing with that activity during free play. In 
contrast, verbal praise increased intrinsic motivation because it drew children's attention to 
their competence. However, praise should not be given indiscriminately or too often or it 
loses its value (Gottfried, 1983). 

Interactivity 

The BORS defines interactivity in terms of reciprocal social interactions such as 
engaging in turn-taking conversation with elaboration. 

Patrick (1997), Schickendanz (1994), and Stone (1993) noted that there was a 
strong association between students' social competence and academic performance, 
especially in the areas of school adjustment, achievement, and motivation. In other 
words, socially popular children are also likely to be academically successful children. 

The well-socialized and socially accepted student is one who is comfortable with his peers 
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and adults, can express feelings and recognize the feelings of others (empathy), knows 
how to share time, space, and materials, and abides by classroom norms, rules, and 
expectations. Students who act cooperatively with peers, respond appropriately to 
teacher directions, participate willingly in class activities, and recognize that different 
behaviors are acceptable for different situations are the socially responsible students who 
pave the way for their own learning, and, by the good example they set, for learning by 
their peers (Patrick, 1997; Stone, 1993). 

Social competence in the school setting implies successful adjustment to school, 
involvement in academic processes, and a commitment to learning. Students who make a 
successful adjustment to school are not likely to drop out or engage in counterproductive 
and anti-social behaviors while in school. Yet if students do not successfully adjust to the 
school setting and, consequently, are rejected or shunned by their classmates, they are 
almost by definition, more likely to avoid school or, if forced to attend, disrupt the 
classroom or be inattentive to what goes on in the classroom. Thus, the student who 
cannot engage productively and competently in social situations is also not likely to be in a 
position to learn academically (Patrick, 1997; Schickendanz, 1994). 

Patrick (1997) noted that not only was low social acceptance strongly related to 
poor school achievement but it was also one of the strongest predictors of school dropout. 
By contrast, children who perceived themselves to be socially acceptable or, better still, 
were really popular, were also motivated to learn. For the youngest children, in fact, their 
own perceptions of their social competence seemed to be a stronger predictor of 
motivation to learn than their own or teachers' perceptions of academic performance 
(Stone, 1993; Patrick, 1997). 
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Quay and Jarret (1986) found in a study of social reciprocity among handicapped 
and nonhandicapped students that social competence or lack of it significantly affected the 
learning process. Because handicapped students were less likely to possess competent 
social skills, they were more likely to be rejected by peers in a mainstream setting. Yet, if 
they learned good social reciprocity from their nonhandicaped peers, their social 
acceptance improved; they felt more comfortable in the school setting, and they finally 
were more receptive to academic learning. 

One important purpose of mainstreaming handicapped children is to teach them by 
the example of their nonhandicapped peers good social skills that, in turn, may promote 
academic learning (Quay & Jarrett, 1986). While some studies have suggested that if 
handicapped children were socially rejected in the mainstream setting, they might be more 
comfortable with other handicapped children, Quay and Jarrett (1986) observed that 
handicapped children could learn reciprocal social skills more effectively from peers who 
already possessed these skills. 

Receptivity 

The BORS defines receptive learners showing openness toward learning and to 
being influenced by teachers/parents/therapists. More specifically, at three years old, 
receptive learners are relatively socialized, able to interact with peers and adults, are not 
excessively dependent on teachers or other adults, usually approach new experiences 
willingly, usually can cope with rejection or setbacks relatively calmly, are capable of some 
objective observation, can empathize with others, and can even show a burgeoning sense 
of humor (McClellan & Katz, 1993; Stone, 1995). 
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By four and five years old, receptive learners, exhibit great curiosity about their 
environment and others in it, and increasingly imitate behaviors of peers and teachers or 
other caregivers/instructors. Four and five old receptive learners are capable of 
conceptual learning, classification skills, operational logic, and symbolic or abstract 
thought. At these ages, receptive learners are also interested in sharing and exchanging 
information, asking questions and demonstrating an interest in good answers (often 
followed by still more questions), and participating in ongoing discussions of any topic 
(Edwards & Springate, 1995; McClellan & Katz, 1993; Stone, 1995). 

The nonreceptive young learner tends to act as if he were neglected by adults, 
isolated by peers, bored with ongoing activity (especially if not directly focused on himself 
and his immediate wants), suspicious of or openly hostile to new experiences or 
information, indifferent towards others instead of curious about his environment or towards 
any instruction, and obviously discouraged when faced with any setbacks or rejection 
(Katz, 1993). 

Perhaps what most distinguishes the receptive young learner from her peers who 
are resistant to learning is that the receptive learner takes pride in demonstrating her 
competency at learned tasks or skills. The resistant young child displays no interest in 
developing a sense of accomplishment (Edwards, Gandini & Nimmo, 1994; Katz, 1993; 
McClellan & Katz, 1993). 

There are a variety of ways to help children become more receptive learners, 
including exposure to words (reading and being read to); involvement in gestural and in 
kinesthetic expression (dramatic and symbolic play); engagement in drawings, paintings, 
sculpture, construction, music, dance; and environmental exploration. Games, group 
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projects and open-ended discussions, whole-language, science, social studies, and 
creative activities and further ways to encourage the receptive learner and coax the 
reluctant or resistant learner (Edwards & Springate, 1995; Stone, 1995). 

The Italian Reggio Emilia preschool model of education (Edwards, Gandin & 
Nimmo, 1994) has found storytelling to be a particularly effective way of engaging whole 
classrooms of preschool children, because storytelling enables children to express their 
interests and emotions through their own stories and dramatizations, while developing 
related projects, as well as acquiring pre-literacy skills. However, storytelling projects 
must include outside input, as well as children's contributions (Edwards et al., 1994). 

Even so, like dramatic play, storytelling in a collaborative context between teacher 
and students promotes among young children the social cooperation and sharing of ideas 
essential to developing receptive learning attitudes. At the Massachusetts Institute of 
Technology, computers are used to help young children design their own storytellers. 
Called SAGE (Storytelling Agent Generation Environment), the program allows children to 
design storyteller agents, create their own database of stories, and interact with their 
stories. Storytelling can be built around specific themes, which, in turn, promote 
collaborative learning, including the interaction of talking, reading and creating, all of which 
promote learning receptivity (Freeman & Sokoloff, 1995; Umaschi, 1996). 

In summary, the studies reviewed provide evidence for the relationship between the 
behaviors that are rated by the BORS and both academic functioning and general 
development. Further, the behaviors included on the BORS are susceptible to 
improvement by experience and instruction. Finally, the behaviors show considerable 
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interaction with each other, in the service of promoting competence in a variety of learning 
situations. 



Purpose of the Study 

Traditional assessment, which uses psychometric measures, is limited in its ability 
to assess the learning potential of children. Instead, dynamic assessment, in which the 
examiner interacts with the child as both tester and facilitator, offers the potential of 
determining not only a child's present abilities but emerging levels of development as well. 
Moreover, because dynamic assessment is inherently interactive, the examiner can make 
observations and inferences that are far more relevant to the classroom setting than 
scores from standardized psychometric measures. 

Even so, some systematic measurement is needed, not only to enable comparing 
an individual child’s functioning to peers but also to monitor the student's progress. The 
Application of Cognitive Functions Scale with its subcomponent of the Behavior 
Observation Rating Scale has been designed as a tool for interactive dynamic 
assessment of young children. Yet it is a new procedure and, as such, its validity and 
reliability have yet to be fully documented. In particular, because behaviors are 
subjectively observed and assessed, the Behavior Observation Rating Scale needs further 
study to explore its validity and reliability. 

In any assessment conducted by a school psychologist, inferences are made 
based on time-limited samples of performance and behaviors in classroom and home 
settings. The validity and reliability of these inferences are rarely directly addressed. This 
study investigated not only psychometric issues related to BORS, but to the assessment 
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situation in general. This study looked at the relationship between the time-limited 
samples of ratings of behaviors by an outside assessor and ratings of the same behaviors 
based on ongoing experience with the child by those working directly with the child, 
namely, the teacher and speech therapist. 

Conclusions from the study can be generalized to other situations in which 
psychologists observe children for relatively brief period of time, and make inferences from 
these observations to more general classroom behaviors. 

Hypotheses 

1. There will be a significantly positive correlation between the experimenter's observation 
of students during a classroom lesson and the classroom teacher's scores of the student's 
general classroom behavior using the parameters of the ACFS Behavior Observation 
Rating Scale. 

2. There will be a significant positive correlation between the ACFS Behavior Observation 
rating Scale scores of the researcher's observations of the children during one speech 
therapy session and the speech therapist's ratings of the child's general behavior during 
speech therapy. 

3. There will be a significant positive correlation between the ratings of the child by the 
teacher and speech therapist. 

4. There will be inter-rater agreement between the researcher and the ACFS principal 
author of 80 percent or above on each of the components, as well as the total score. 
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Methodology 

Participants 

Participants in this study were from the Hebrew Academy for Special Children. The 
sample included 25 children from four classes in the preschool setting, of whom 70% were 
Jewish and the rest Asian or Hispanic. Four teachers and seven speech therapists 
participated in this study. The children's ages ranged from two and a half to five year old, 
with the mean of 3.8. The children enrolled in the school exhibited various developmental 
delays such as speech disorders, cognitive delays, and behavioral disorders such as 
ADHD. In addition, some children were diagnosed with Downs' Syndrome and Pervasive 
Developmental Disorders and multiple handicapped conditions. All the participants in this 
study were eligible for preschool special education services. 

The children lived in the Brooklyn and Long Island area in close proximity to the 
school. The socioeconomic background of the children's families varied greatly. Some of 
the children were from affluent homes; many were middle class and others were welfare 
recipients. Many children in this sample lived with their parents in bilingual 
English/Hebrew speaking homes. Some children lived in a non-residential setting. All 
teachers held master's degrees in Special Education, and the speech therapists held 
Master’s degrees in Speech Therapy. The program provides comprehensive services 
including speech, occupational, and physical therapy. The class size varied from eight to . 
ten children. Age, type of disability, and the severity of the handicapping condition 
determined the assignments of children to the classes. Each class had a teacher and two 
paraprofessionals, and in some cases the children required a one-on-one aide. 
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Each child in this study received biweekly speech therapy session, and each has been 
involved in speech therapy for a minimum of one year. 

Measure 

The Application of Cognitive Functions Scale (ACFS) Behavior Observation Rating 
Scale consists of seven components: 

1) Self-Regulation: the extent of the child's ability to self-regulate and /or inhibit impulsive 
responding. 

2) Persistence: the extent of the child’s persistence to complete the task. 

3) Frustration Tolerance: the child’s ability to regain compliance when experiencing 
frustration related to task difficulty. 

4) Flexibility: the child's attempts towards alternative solutions or self-correction while 
solving tasks. 

5) Motivation: the extent of the child's affective response/reaction or interest in task or 
materials. 

6) Interactivity: the extent of the child's reciprocal social interaction. 

7) Receptivity: the extent of the child's openness to experiencing intervention by the 
mediator. 

The ACFS Behavior Rating Scale describes the child's behavior during the 
administration of each subtest of the ACFS assessment. This scale has been abstracted 
from the ACFS and modified for application to classroom and therapy session observation 
by the ACFS first author. One version was for the use of the researcher, to ease 
recording of observation; only the recording format was altered. Another version was 
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created to facilitate completion by teachers and therapists. On both adaptations, the 
components are identical to the original scale. Each item has a rating from 1 to 3 ranging 
from no evidence (1 ) to optimal (3). The teacher's adapted form of the scale summarizes 
the components in one sentence and has a similar rating hierarchy. Each version of the 
scale yields a total possible score of 21 (range of 7-21). In this form the ratings were V 
(very much) S (sometimes) N (not at all). For purposes of calculation V equals 3, S 
equals 2 and N equals 1. This scoring system is consistence with original form. 

Two previous studies included findings that translated to the current research with 
the BORS. Shurin’s (1998) Master’s thesis involved 26 four-year-old children. The study 
involved the Behavior Rating Scale that is embedded in each of the ACFS subtests. The 
first question explored the relationship between the behavior-rating total ACFS scores and 
total task ACFS scores: Does the child’s behavior during the procedure relate to task 
competence? The correlation of .65 (p <.001 ) offered a qualified yes. Specific behaviors 
that related to task competence were Persistence, Frustration Tolerance and Flexibility. 
The study also provided evidence regarding intratest reliability of the BORS, documenting 
a significant correlation of each component to total BORS score. 

Another attempt to evaluate interscore T reliability involved two independent raters 
(certified school psychologists) who attended a workshop offered by Lidz (in preparation). 
The attendees of the workshop were introduced to dynamic assessment and heard an 
overview of the ACFS that included the Behavior Observation Rating Scale. During a live 
demonstration of the ACFS, two participants volunteered to rate the child along with the 
author. Thus the raters had very minimal exposure to the scale. Nevertheless, the 
average rate of agreement among all raters was 74% and the behavior component that 
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elicited the highest levels of agreement were Persistence 100%, Frustration Tolerance 
94%, and Motivation 78%. 

Levy’s (1999) study of interscore reliability check of the BORS also earned the 
exact agreement of ratings for the pretest was 77% and for the mediations 70%. The 
percentage of total agreements across all tasks was 67%. 

Procedures 

Materials for this study included the ACFS Behavior Observation Rating Scale in 
both questionnaire and rating scale format (Appendixes A & B), and letters of consent 
from administration and staff. The researcher secured permission to conduct the study 
from the program administrators. Since this was a non-intrusive study, no permission for 
individual students was necessary. However, the agency secured general permission for 
assessment, which is in each student's file. The researcher met with the teachers and 
speech therapists to solicit their cooperation, introduce them to the scales, and arrange for 
observation times. 

The researcher asked the children’s classroom teachers to complete the teacher 
version of the ACFS - BORS (Appendix A) based on their observations of the children's 
general classroom behavior. The researcher then observed and rated with the original 
ACFS - BORS (Appendix B) each child's participation during two classroom tasks: free 
play and circle time activities. The researcher observed a speech therapy session of each 
child participant and rated with the original ACFS BORS (Appendix B). The researcher 
then asked the speech therapist to complete the teacher/therapist version of the scale, 
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rating the child's behavior on the scale based on her observations of the children’s general 
behavior during the speech therapy sessions. 



Results 

The first two hypotheses addressed the validity and the second two the reliability of 
the BORS. 

The first hypothesis stated that there would be a significantly positive correlation 
between the experimenter’s ratings and the mean score of all teacher’s rating of the 
BORS. This relationship was analyzed using a Pearson r. The Cronbach Alpha for the 
six components observed by researcher was .69 and for the teachers’ ratings was .78. 

Table 1 - Correlation between Researcher’s and Teachers’ (n = 4) ratings 



Self-regulation .81*** 

Persistence .74*** 

Flexibility .58** 

Motivation .83*** 

Interactivity 1.00*** 

Receptivity .76*** 

Total .95*** 



p< .01 ***p< 001 






Data from Table 1 support this hypothesis. Table 1 shows a statistically significant 
positive correlation between the examiner and teachers’ ratings of each component, and 
the total. In this analysis Frustration Tolerance was not included because the researcher 
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was not able to obtain a score on that component for many of the children. Four of the 
seven coefficients exceeded .80. 

The second hypothesis stated that there would be a significant positive correlation 
between the researcher’s ratings and the mean score of all speech therapists’ ratings of 
students on the BORS. This was analyzed by using the Pearson r and results appear in 
Table 2. 

The Cronbach Alpha for the six components rated by the researcher was .77 and 
for the speech therapist ratings was .81. 



Table 2 — Correlation between Researcher’s and Speech Therapists’ (N=7) Ratings 



Self-regulation 

Persistence 

Flexibility 

Motivation 

Receptivity 

Total 



1 . 00 *** 

88 *** 

88 *** 

.78*** Interactivity 
.83*** 

gy*** 



1 . 00 *** 



***p< 001 

Table 2 shows that the relationship between the researcher’s and speech therapist’ 
ratings were all highly statistically significant, with ratings on Self-Regulation and 
Interactivity reaching perfect agreement. Six of the seven coefficients exceeded .80. 

In this analysis Frustration Tolerance was not included because the researcher was 
not able to obtain a score on that component for many of the children. 

The third hypothesis stated that there would be a significantly positive correlation 
between the mean scores of the teacher’s ratings and the mean score of all speech 
therapists ratings of the children on the BORS. This was analyzed by using the Pearson r, 
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and results appear in Table 3. The Cronbach Alpha for the seven components rated by 
teachers was .79 and for speech therapist was .81 . 

Table 3 - Correlation between Speech therapists (N=7) and teachers (N=4) 

Self-regulation .90*** 

Persistence .62** 

Frustration Tolerance .88*** 

Flexibility .62** 

Motivation .82*** 

Interactivity .92*** 

Receptivity .83*** 

Total .94*** 

** p< .01 ***p<. 001 

Table 3 shows that all BORS ratings by teachers and speech therapists on all 
components reached a high level of agreement, with Six of the ei^hTt components 
exceeding .80. All seven components were included in this study. 

For the first three hypotheses there were correlations. Therefore it was 

necessary to apply the Bonferroni method, which adjust the statistical criterion for 
significance to .001 . 

The fourth hypothesis states that there will be an inter-rater agreement between the 
researcher and ACFS principal author of 80 percent on each component as well as the 
total score. 
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Table 4 - Inter rater Agreement between Researcher’s and Author’s ratings 

COMPONENTS % AGREEMENT 

Self-regulation 67 

Persistence 33 

Frustration Tolerance 100 

Flexibility 0 

Motivation 100 

Interactivity 100 

Receptivity 100 

Total Average Level 71 

Of agreement 

Data from Table 4 show level of agreement between researcher and author on 
each of the seven components. Four of the seven components show 1 00% of agreement: 
Frustration Tolerance, Motivation, Interactivity and Receptivity. For the component of 
Persistence the author and researcher only agreed on 1 out of the three cases and on 
Flexibility there was no agreement. Overall, there was an inter-rater agreement of 71%. 



Discussion and conclusion 

This study sought to examine aspects of the validity and reliability of the AFCS- 
BORS in relation to both the parameters of the BORS, and in terms of the implications of 
the results of the study for the relationship between psychological assessment and 
classroom performance. 

The results showed that there were highly significant positive correlationsbetween 
the researcher’s ratings behavior of students engaged in a sample classroom activity and 
their teachers’ rating of children’s general behavior. 
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There was also a significant positive correlation between the researcher’s ratings of 
the children during a speech session and their speech therapist’s ratings of the child’s 
general behavior. Informal comparison of the ratings of classroom and speech therapy 
session suggesfcgenerally higher correspondence between the researcher and the speech 
therapist, which may reflect the greater easeof rating the children in the one to one 
situation of speech therapy. There was a significantly high correlation between ratings of 
the children by teachers and speech therapists. Ttus supports the stability of the 
children’s behavior across the two situations, as well as the reliability of the scale across 
situations, despite ratings by a variety of observes. 

The wide range of inter-rater agreement between the researcher and the ACFS 
principal author suggests the need to interpret these results with caution, as well as the 
need for further research in this area. /tnoxh^reason why the results obtained did not 
fully support the hypothesis could be attributecfto the fact that only three cases of inter- 
rater agreement were used in this study. Also, two out of the three cases the researcher 
could not rate flexibility since it was not applicable, so the results were based on one case 
and there was no agreement for the component. ' 
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There were other limitationsof this study,- as a correlational study, it could only 
provide data regarding the degree of relationship between variables listed by the BORS. 
Thus, interpretation of the results is limited by the inability of the study to separate 
reliability of the scale from reliability of the behaviors tapped by the scale. 

Other limitations concern the sample. Because the sample size was necessarily 
small (25) and limited to students with special needs, it is arguable that the results cannot 
be readily generalized to a larger and more diverse population, including a broader range 
of ethnicity's and subcultures. 

The BORS ratings are also vulnerable to rater biases, particularly those with 
negative and positive halo effects, despite orientation of the raters by the researcher to 
minimize this. The limited number of raters made it difficult to separate scale from rater 
effects, especially because there were more speech therapists available than teachers at 
the time this study was conducted. 

Given these limitation^ the present study results not only largely supported the 
hypotheses but also compared favorably with other studies of the BORS. For example, 
Shurin (1999) selected a case for an interscorer reliability check of the Behavior 
Observation Rating Scale, comparing the scores assigned by Shurin and Lidz (1999). 

The percent of exact agreement for ratings during the pretest was 74% and 82% for 
ratings during mediation. When the ratings for pretest and mediation were combined, and 
the number of instances of agreements and disagreements for each behavior components 
across all tasks were compare, the percentage of agreement was 96%. A similar 
comparison was done in relation to another thesis (Levy, 1999). In this case, the exact 
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agreement of ratings for the pretest was 77%, and, for the mediations, 70%. The 
percentage of total agreements across all tasks was 67%. 

Moreover, the present study, as well as other studies of the BORS scale (Lidz & 
Jepsen, 1997; Lidz, 1995) found considerable interaction and interrelationship among 
various components of behaviors measured on the BORS scale, thus extending the 
findings of other literature on those behaviors. For example, the high levels of agreement 
among testers in the present study for persistence, frustration tolerance, and motivation 
suggest high level of correlation among these behaviorsyv^s also noted in the literature. 
Stone (1993) found that frustration tolerance seemed to be the basis for all intellectual 
effort but especially for motivation and persistence. On the other hand, .. Nelson-LeGall 
& Scott-Jones (1985) found too much frustration tolerance and persistence can lead to 
repetitive, unproductive behaviors and therefore must be leavened with high levels of 
flexibility. 

Flexibility, in turn, correlates with interactivity, as shown by Patrici(1997) and 
Sch(cK£n4‘ 3 (3994), who found that students who were flexible enough to adjust their 
behaviors to varying situations were the social and academic leaders in their classrooms. 
Schickedanz (1994), as well as Nelson-LeGall & approaches correlated with high 
frustration tolerance levels, whereas full completion of tasks seemed to be most readily 
achieved by the highest motivated children (Stone, 1993). 

The BORS scale’s ability to v • measure all these behaviors and show the 
correlation s among them makes it useful not only as a predictor of educational success 
(or lack of success) but as guide towards targeting individual children’s specific problem 
areas. 
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Psychologists assume that the results of their assessment are generalizable to 
situations outside of the assessment context. In the case of school psychologist} it f s 
important that assessment results are generalizable to the classroom and other 
instructional settings of the student. Therefore, it is encouraging to find the strong 
relationship between the researcheisbehavioral rating and those of both the teachers and 
the speech therapists, since the researcher’s observations on a much more limited time 
sample. Thus, this study helped to demonstrate the usefulness of the BORS as an 
assessment aide for school psychologists seeking indicators of those behaviors that 
contribute to learning exhibited by young children both in the classroom and individually 
when removed from the classroom for individual assessment. The v <xri assessed 
by the BOR s can provide a common language and frame of reference for psychologist^ 
teachers, and therapists who provide services to the same children. 

Possibilities for further study might include testing the BORS with a larger number 
and variety of examiner/raters and with a more diverse student population, including 
regular education students. 
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Graduate School of Education & Psychology 
TOURO COLLEGE 



ACFS Behavior Observation Rating Scale 

TEACHER/THERAPIST FORM 

(C)l998 Lidz/Jepsen 



Dear Teacher or Speech Therapist, 

Please rate the child indicated on this form on the following items. The rating 
should reflect the child’s typical behavior when you are trying to teach this child 
something new. You may add any comments you wish to explain or elaborate your 
ratings. Be sure to consider each item separately so that you avoid giving the same rating 
for each. Thank you very' much for your time. 

Child: Rater: Location: 

Date: How long have you known this child? 

3 ‘ a i 

Very much Sometimes Not at all 
like this child like this child like this child 



1 . The child show's aood self control. 



V S N 



2. The child completes all tasks. V S 

3. When frustrated, the child recoups quickly. V S 

4. The child is flexible in solving problems 

and doesn’t get stuck on one approach. V S 

5. The child shows interest in and enthusiastic 

reactions to materials and tasks. V S 



N 

N 



N 



N 



6. The child is able to participate in and take 

turns in conversations. V 



S 



N 



7. The child is a walling learner. 



V 



S 



N 
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ACFS Behavior Observation Rating Scale 
C 1998 Lidz/Jepsen 



Child’s name: 




Date of Observation: 


Rater: 


Location: 


Activity: 



For each activity observed, select one rating per behavior described below: 

Behavior Rating 



Notes 



SELF REGULATION: regulates attention and inhibits impulsivity. 



• Maintains attention and controls impulsivity 3 

• Requires mild intervention from adult 2 

• Requires significant intervention from adult 1 

PERSISTENCE: completes task or activity 

• Completes without seeking to terminate 3 

• Completes activity with encouragement 2 

• Quits and cannot be reengaged 1 



FRUSTRATION TOLERANCE: when frustrated, recoups and continues 

• When upset, is easily calmed and redirected 3 

• When upset, eventually calms down and reengages 2 

• When upset, cannot be calmed 1 

FLEXIBILITY: tries alternative solutions or approaches 

• Readily changes approach or self corrects 3 

• Attempts alternative, but new attempt is similar to first 2 

• Makes no change; gets stuck on initial attempt or approach I 

MOTIVATION: shows positive affective response to and interest in activity 

• Shows enthusiastic response to activity 3 

• Shows neutral reaction, but proceeds without protest 2 

• Shows little or negative reaction; may state dislike 1 

INTERACTIVITY shows reciprocal social interactions 

• Engages in turn-taking conversations with elaboration 3 

• Engages in turn-taking conversations with minimal responses 2 

• Engages in no turn-taking conversations 1 

RECEPTIVITY: shows openness to learning and being influenced by the teacher/therapist 

• Isa willing and receptive learner 3 

• Insonsistentlv willing and receptive 2 

• Quite resistant to learning 1 



TOTAL: 
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